Aggregation-Based Learning in the Inverted Pendulum Problem

We consider the problem of adapting approximate dynamic programming techniques to the inverted pendulum task. This is a particularly challenging task as we work with a relatively uninformative reinforcement signal and have no a priori information about our system. Success in this task requires an effective solution to the credit assignment problem, incorporation of noisy and biased information into our belief about the system, and efficient learning. We use an aggregation-based Bayesian prior to estimate our value function and explore the performance of the knowledge gradient algorithm relative to other policies. We deal with the credit assignment problem through the use of a decaying trace of the reinforcement signal. Although this updating mechanism violates some assumptions of traditional learning models, we find that the knowledge gradient policy is effective in improving performance.